<!DOCTYPE html>
<html lang="zh">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
        <title>cs229 第四节 | Dirac Lee</title><meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="robots" content="noodp" />
<meta name="Description" content="cs229 第四节"><link rel="prev" href="https://diraclee.gitee.io/2020/07/cs229_lecture3/" /><link rel="next" href="https://diraclee.gitee.io/2020/07/cs229_lecture5/" /><link rel="canonical" href="https://diraclee.gitee.io/2020/07/cs229_lecture4/" />
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff"><meta property="og:title" content="cs229 第四节" />
<meta property="og:description" content="cs229 第四节" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://diraclee.gitee.io/2020/07/cs229_lecture4/" />
<meta property="article:published_time" content="2020-07-21T00:00:00+00:00" />
<meta property="article:modified_time" content="2020-07-22T00:00:00+00:00" />
<script type="application/ld+json">
    {
        "@context": "http://schema.org",
        "@type": "BlogPosting",
        "headline": "cs229 第四节",
        "mainEntityOfPage": {
            "@type": "WebPage",
            "@id": "https:\/\/diraclee.gitee.io\/2020\/07\/cs229_lecture4\/"
        },"image": {
                "@type": "ImageObject",
                "url": "https:\/\/diraclee.gitee.io\/logo.png",
                "width":  800 ,
                "height":  600 
            },"genre": "posts","keywords": "机器学习, cs229","wordcount":  1097 ,
        "url": "https:\/\/diraclee.gitee.io\/2020\/07\/cs229_lecture4\/","datePublished": "2020-07-21T00:00:00\x2b00:00","dateModified": "2020-07-22T00:00:00\x2b00:00","license": "This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.","publisher": {
                "@type": "Organization",
                "name": "xxxx",
                "logo": {
                "@type": "ImageObject",
                "url": "https:\/\/diraclee.gitee.io\/logo.png",
                "width":  127 ,
                "height":  40 
                }
            },"description": "cs229 第四节"
    }
    </script><link rel="stylesheet" href="/css/style.min.css"><link rel="stylesheet" href="/css/lib/fontawesome-free/all.min.css"><link rel="stylesheet" href="/css/lib/animate/animate.min.css"></head>
    <body><script>
            window.isDark = (window.localStorage && window.localStorage.getItem('theme')) === 'dark';
            window.isDark && document.body.classList.add('dark-theme');
        </script><div class="wrapper"><nav class="navbar">
    <div class="navbar-container">
        <div class="navbar-header animated bounceIn">
            <a href="https://diraclee.gitee.io">Dirac Lee</a>
        </div>
        <div class="navbar-menu"><a class="menu-item" href="https://diraclee.gitee.io/posts" title="">文章</a><a class="menu-item" href="https://diraclee.gitee.io/tags" title="">标签</a><a class="menu-item" href="https://diraclee.gitee.io/categories" title="">分类</a><a class="menu-item" href="https://diraclee.gitee.io/about" title="">关于</a><a class="menu-item" href="https://diraclee.gitee.io" title="English"><i class="fas fa-language fa-fw"></i></a><a href="javascript:void(0);" class="theme-switch"><i class="fas fa-adjust fa-rotate-180 fa-fw" title="切换主题"></i></a>
        </div>
    </div>
</nav><nav class="navbar-mobile">
    <div class="navbar-container">
        <div class="navbar-header">
            <div class="navbar-header-title animated bounceIn">
                <a href="https://diraclee.gitee.io">Dirac Lee</a>
            </div>
            <div class="menu-toggle" id="menu-toggle">
                <span></span><span></span><span></span>
            </div>
        </div>
        <div class="navbar-menu" id="mobile-menu"><a class="menu-item" href="https://diraclee.gitee.io/posts" title="">文章</a><a class="menu-item" href="https://diraclee.gitee.io/tags" title="">标签</a><a class="menu-item" href="https://diraclee.gitee.io/categories" title="">分类</a><a class="menu-item" href="https://diraclee.gitee.io/about" title="">关于</a><a class="menu-item" href="https://diraclee.gitee.io" title="English"></a><a href="javascript:void(0);" class="theme-switch"><i class="fas fa-adjust fa-rotate-180 fa-fw" title="切换主题"></i></a>
        </div>
    </div>
</nav><main class="main">
                <div class="container"><article class="page"><h1 class="post-title animated flipInX">cs229 第四节</h1><div class="post-meta">
            <div class="post-meta-main"><a class="author" href="https://diraclee.gitee.io" rel="author" target="_blank">
                    <i class="fas fa-user-circle fa-fw"></i>Dirac Lee
                </a>&nbsp;<span class="post-category">收录于&nbsp;<i class="far fa-folder fa-fw"></i><a href="https://diraclee.gitee.io/categories/%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/">学习笔记</a>&nbsp;</span></div>
            <div class="post-meta-other"><i class="far fa-calendar-alt fa-fw"></i><time datetime=2020-07-21>2020-07-21</time>&nbsp;
                <i class="fas fa-pencil-alt fa-fw"></i>约 1097 字&nbsp;
                <i class="far fa-clock fa-fw"></i>预计阅读 3 分钟&nbsp;</div>
        </div><div class="post-content"><ul>
<li>感知机</li>
<li>指数家族</li>
<li>广义线性模型(GLMs)</li>
<li>Softmax 回归(多分类问题)</li>
</ul>
<a class="post-dummy-target" id="感知机"></a><h2>感知机</h2>
<p>$h_\theta (x) = g(\theta^T x)$ ，其中 $g(z) = \begin{cases} 1 &amp; z \ge 0 \\  0 &amp; z &lt; 0 \end{cases}$</p>
<p>参数更新方式 
$$
\theta_j := \theta_j + \alpha (y^{(i)} - h_\theta (x^{(i)})) x_j^{(i)}
$$</p>
<ul>
<li>当 $y^{(i)} - h_\theta (x^{(i)}) = 0$ 时，预测正确，不更新 $\theta$</li>
<li>当 $y^{(i)} - h_\theta (x^{(i)}) = 1$ 时，预测错误，预测值为 $h_\theta (x^{(i)}) = 0$ 正确值为 $y^{(i)} = 1$ ，因此增大 $\theta$ 从而使 $h_\theta (x^{(i)})$ 更趋近于 1</li>
<li>当 $y^{(i)} - h_\theta (x^{(i)}) = -1$ 时，预测错误，预测值为 $h_\theta (x^{(i)}) = 1$ 正确值为 $y^{(i)} = 0$ ，因此减小 $\theta$ 从而使 $h_\theta (x^{(i)})$ 更趋近于 0</li>
</ul>
<a class="post-dummy-target" id="指数家族-ef"></a><h2>指数家族 (EF)</h2>
<a class="post-dummy-target" id="定义"></a><h3>定义</h3>
<p>概率密度函数 
$$
P(y; \eta) = b(y) \exp(\eta^T T(y) - a(\eta)) 
$$</p>
<p>其中</p>
<p>$y$ - 数据</p>
<p>$\eta$ - 自然参数</p>
<p>$T(y)$ - 充分统计量</p>
<p>$b(y)$ - 基本度量</p>
<p>$a(\eta)$ - 对数分割函数</p>
<a class="post-dummy-target" id="伯努利分布-二分类问题"></a><h3>伯努利分布 (二分类问题)</h3>
<p>$$
\begin{aligned}
P(y; \phi) 
&amp;= \phi^y (1 - \phi)^{1 - y}  \\ 
&amp;= \exp[log(\phi^y (1 - \phi)^{1 - y})] \\ 
&amp;= 1 \exp [  \log (\frac \phi {1 - \phi}) y  + \log (1 - \phi)] \\ 
&amp;= b(y) \exp(\eta^T T(y) - a(\eta))
\end{aligned}
$$</p>
<p>其中</p>
<ul>
<li>$b(y) = 1$</li>
<li>$\eta = \log (\frac \phi {1 - \phi})$  即 $\phi = \frac 1 {1 + e^{-\eta}}$ 这也就是为什么逻辑回归选择 sigmoid 函数的原因</li>
<li>$T(y) = y$</li>
<li>$a(\eta) = - \log (1 - \phi) = -\log(1 - \frac 1 {1 + e^{-\eta}}) = \log (1 + e^\eta)$</li>
</ul>
<p>因此伯努利分布属于指数家族</p>
<a class="post-dummy-target" id="高斯分布回归问题"></a><h2>高斯分布(回归问题)</h2>
<p>为了简化推导，假设方差固定为 $\sigma^2 = 1$</p>
<p>$$
\begin{aligned}
P(y; \phi) 
&amp;= \frac 1 {\sqrt {2 \pi}} \exp(- \frac {(y - \mu)^2} 2) \\ 
&amp;= \frac 1 {\sqrt {2 \pi}} e^{- \frac {y^2} 2} \exp(\mu y - \frac 1 2 \mu^2) \\ 
&amp;= b(y) \exp(\eta^T T(y) - a(\eta))
\end{aligned}
$$</p>
<p>其中</p>
<ul>
<li>$b(y) = \frac 1 {\sqrt {2 \pi}} e^{- \frac {y^2} 2}$</li>
<li>$\eta = \mu$</li>
<li>$T(y) = y$</li>
<li>$a(\eta) = \frac 1 2 \mu^2 = \frac 1 2 \eta^2$</li>
</ul>
<p>因此高斯分布属于指数家族</p>
<p>指数家族的性质</p>
<ol>
<li>关于 $\eta$ 的 MLE 是 concave 函数<br>
关于 $\eta$ 的 NLL 是 convex 函数</li>
<li>平均值 $E[y; \eta] = \frac \partial {\partial \eta} a(\eta)$</li>
<li>方差 $Var[y; \eta] = \frac {\partial^2} {\partial \eta^2} a(\eta)$</li>
</ol>
<a class="post-dummy-target" id="数据的分布"></a><h3>数据的分布</h3>
<table>
<thead>
<tr>
<th>数据范围</th>
<th>概率分布</th>
<th>注解</th>
</tr>
</thead>
<tbody>
<tr>
<td>实数</td>
<td>高斯分布</td>
<td></td>
</tr>
<tr>
<td>二分类 ( 0, 1 )</td>
<td>伯努利分布</td>
<td></td>
</tr>
<tr>
<td>数量 ( 正整数 )</td>
<td>泊松分布</td>
<td></td>
</tr>
<tr>
<td>正实数</td>
<td>Gamma, Exponential</td>
<td></td>
</tr>
<tr>
<td>数据本身是概率分布</td>
<td>Beta, Dirichlet</td>
<td>Bayesian</td>
</tr>
</tbody>
</table>
<a class="post-dummy-target" id="广义线性模型-glms"></a><h2>广义线性模型 GLMs</h2>
<a class="post-dummy-target" id="设计选择"></a><h3>设计选择</h3>
<p>(1) $y|x; \theta \sim \mathbf {EF}(\eta)$<br>
(2) $\eta = \theta^T x$，其中 $\theta \in \R^n$，$x \in \R^n$<br>
(3) 验证阶段，输出 $E[y; \eta]$，即 $h_\theta (x) = E[y; \eta]$</p>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/20200722103404.png" alt="" class="lazyload"></figure></p>
<a class="post-dummy-target" id="训练过程"></a><h3>训练过程</h3>
<p>广义线性模型都按照以下形式进行参数更新</p>
<p>$$
\theta_j := \theta_j + \alpha (y^{(i)} - h_\theta (x^{(i)})) x_j^{(i)}
$$</p>
<a class="post-dummy-target" id="术语"></a><h3>术语</h3>
<p>$\eta$ - 自然参数<br>
$\mu = E[y|\eta] = g(\eta) = \frac \partial {\partial \eta} a(\eta)$ - 规范响应函数<br>
$\eta = g^{-1}(\mu)$ - 规范链接函数</p>
<a class="post-dummy-target" id="三种参数化方式"></a><h3>三种参数化方式</h3>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/20200722105839.png" alt="" class="lazyload"></figure></p>
<a class="post-dummy-target" id="举例"></a><h3>举例</h3>
<a class="post-dummy-target" id="线性回归"></a><h4>线性回归</h4>
<p>线性回归是服从高斯分布的广义线性模型</p>
<p>$$
P(y; \mu) = b(y) \exp(\eta^T T(y) - a(\eta))
$$
其中</p>
<ul>
<li>$b(y) = \frac 1 {\sqrt {2 \pi}} e^{- \frac {y^2} 2}$</li>
<li>$\eta = \mu$</li>
<li>$T(y) = y$</li>
<li>$a(\eta) = \frac 1 2 \mu^2 = \frac 1 2 \eta^2$</li>
</ul>
<p>那么</p>
<p>$$
h_\theta (x) = E[y|x;\theta] = g(\eta) = \frac \partial {\partial \eta} a(\eta) = \eta = \theta^T x
$$</p>
<a class="post-dummy-target" id="逻辑回归"></a><h4>逻辑回归</h4>
<p>逻辑回归是服从伯努利分布的广义线性模型</p>
<p>$$
P(y; \mu) = b(y) \exp(\eta^T T(y) - a(\eta))
$$
其中</p>
<ul>
<li>$b(y) = 1$</li>
<li>$\eta = \log (\frac \phi {1 - \phi})$  即 $\phi = \frac 1 {1 + e^{-\eta}}$</li>
<li>$T(y) = y$</li>
<li>$a(\eta) = - \log (1 - \phi) = - \log(1 - \frac 1 {1 + e^{-\eta}}) = \log (1 + e^\eta)$</li>
</ul>
<p>那么</p>
<p>$$
h_\theta (x) = E[y|x;\theta] = g(\eta) = \frac \partial {\partial \eta} a(\eta) = \frac {e^\eta} {1 + e^\eta} = \frac 1 {e^{-\eta} + 1} = \frac 1 {e^{-\theta^T x} + 1} 
$$</p>
<a class="post-dummy-target" id="softmax回归"></a><h2>Softmax回归</h2>
<p>Softmax回归用于处理多分类问题</p>
<a class="post-dummy-target" id="数据表示"></a><h3>数据表示</h3>
<p>特征 $x \in R^n$，n 为特征维数<br>
标签 $y = [y_{c_1}, y_{c_k}, &hellip;, y_{c_k}]$，k 为分类个数，其中 $\forall y_{c_i} \in \lbrace 0, 1 \rbrace$，
例如 $y = [0, 1, 0, &hellip;, 0]$</p>
<a class="post-dummy-target" id="交叉熵损失"></a><h3>交叉熵损失</h3>
<p>$$
CrossEnt(P, \hat P) = - \sum P(y_{c_i}) \log \hat P(y_{c_i}) 
$$</p>
<p><figure><img src="/svg/loading.min.svg" data-sizes="auto" data-src="https://gitee.com/DiracLee/picbed/raw/master/img/20200722124157.png" alt="" class="lazyload"></figure></p>
<p>上例中，交叉熵损失</p>
<p>$$
CrossEnt(P, \hat P) = - \log \hat P(y_{c_3}) = - \log \frac {e^{\theta_{c_3}^T x}} {\sum_i e^{\theta_{c_i}^T x}}
$$</p>
<p>对交叉熵损失应用梯度下降算法，即可找到合适的模型参数</p>
<a class="post-dummy-target" id="参考"></a><h2>参考</h2>
<p><a href="https://katex.org/docs/supported.html" target="_blank">$\KaTeX$ 文档</a></p></div><div class="post-footer" id="post-footer">
    <div class="post-info">
        <div class="post-info-line">
            <div class="post-info-mod">
                <span>本文于 2020-07-22 更新</span>
            </div>
            <div class="post-info-license"></div>
        </div>
        <div class="post-info-line">
            <div class="post-info-md"></div>
            <div class="post-info-share"><span><a href="//www.linkedin.com/shareArticle?url=https%3a%2f%2fdiraclee.gitee.io%2f2020%2f07%2fcs229_lecture4%2f&amp;title=cs229%20%e7%ac%ac%e5%9b%9b%e8%8a%82" target="_blank" title="分享到 LinkedIn">
            <i class="fab fa-linkedin fa-fw"></i>
        </a><a href="//service.weibo.com/share/share.php?url=https%3a%2f%2fdiraclee.gitee.io%2f2020%2f07%2fcs229_lecture4%2f&amp;appkey=&amp;title=cs229%20%e7%ac%ac%e5%9b%9b%e8%8a%82" target="_blank" title="分享到 Weibo">
            <i class="fab fa-weibo fa-fw"></i>
        </a></span></div>
        </div>
    </div>

    <div class="post-info-more">
        <section><span class="tag">
                        <a href="https://diraclee.gitee.io/tags/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/"><i class="fas fa-tag fa-fw"></i>&nbsp;机器学习</a>&nbsp;
                    </span><span class="tag">
                        <a href="https://diraclee.gitee.io/tags/cs229/"><i class="fas fa-tag fa-fw"></i>&nbsp;cs229</a>&nbsp;
                    </span></section>
        <section>
            <span><a href="javascript:window.history.back();">返回</a></span>&nbsp;|&nbsp;<span><a href="https://diraclee.gitee.io">主页</a></span>
        </section>
    </div>

    <div class="post-nav"><a href="https://diraclee.gitee.io/2020/07/cs229_lecture3/" class="prev" rel="prev" title="cs229 第三节"><i class="fas fa-angle-left fa-fw"></i>cs229 第三节</a>
            <a href="https://diraclee.gitee.io/2020/07/cs229_lecture5/" class="next" rel="next" title="cs229 第五节">cs229 第五节<i class="fas fa-angle-right fa-fw"></i></a></div>
</div><div class="post-comment"></div>
    </article></div>
            </main><footer class="footer">
    <div class="copyright"><div class="copyright-line">由 <a href="https://gohugo.io/" target="_blank" rel="external nofollow noopener noreffer">Hugo</a> 强力驱动 | 主题 - <a href="https://github.com/dillonzq/LoveIt" target="_blank" rel="external nofollow noopener noreffer">LoveIt<i class="far fa-heart fa-fw"></i></a>
        </div>

        <div class="copyright-line"><i class="far fa-copyright fa-fw"></i><span itemprop="copyrightYear">2020</span><span class="author" itemprop="copyrightHolder">&nbsp;<a href="https://diraclee.gitee.io" target="_blank">Dirac Lee</a></span>&nbsp;|&nbsp;<span class="license"><a rel="license external nofollow noopener noreffer" href="https://creativecommons.org/licenses/by-nc/4.0/" target="_blank">CC BY-NC 4.0</a></span></div>
    </div>
</footer></div><a href="#" class="dynamic-to-top" id="dynamic-to-top" data-scroll>
            <span>&nbsp;</span>
        </a><script src="/js/lib/jquery/jquery.slim.min.js"></script><script src="/js/lib/lazysizes/lazysizes.min.js"></script><script src="/js/lib/smooth-scroll/smooth-scroll.polyfills.min.js"></script><script>window.scroll = new SmoothScroll('[data-scroll]', {speed: 300, speedAsDuration: true});</script><link rel="stylesheet" href="/css/lib/katex/katex.min.css"><script src="/js/lib/katex/katex.min.js"></script><script defer src="/js/lib/katex/auto-render.min.js"></script><link rel="stylesheet" href="/css/lib/katex/copy-tex.min.css"><script defer src="/js/lib/katex/copy-tex.min.js"></script><script defer src="/js/lib/katex/mhchem.min.js"></script><script>
        document.addEventListener("DOMContentLoaded", function () {
            renderMathInElement(document.body, {
                delimiters: [
                    { left: "$$", right: "$$", display: true },
                    { left: "\\(", right: "\\)", display: false },
                    { left: "\\[", right: "\\]", display: true },{ left: "$", right: "$", display: false },]
            });
        });
    </script><script src="/js/blog.min.js"></script></body>
</html>